The need to analyze disparate datasets and to utilize different processing paradigms has led to a profusion of distributed cluster frameworks. To consolidate data center resources, combine various processing paradigms within the same application, and facilitate inter-framework data sharing, a number of approaches have been designed that include high-performance computing- (HPC-) style centralized managers, centralized two-level managers, and decentralized managers.
Such existing approaches, however, include multiple disadvantages such as, for example, encompassing a limited temporal scope, failing to utilize available resources on relevant processing nodes, and precluding platforms to optimize work according to platform-specific metrics. Accordingly, there is a need for a cross-platform scheduler which addresses such disadvantages and which will simultaneously provide improved flexibility, performance and fairness.